Audiovisual verbal transformations, as a way to study audiovisual interactions in speech perception

Verbal transformations refer to perceptual changes experienced while listening to a speech form cycled in rapid and continuous repetition. In order to test whether visual information from the speaker's articulatory gestures may modify the emergence and stability of verbal auditory percepts, participants were instructed to report any perceptual changes during unimodal, congruent audiovisual and incongruent audiovisual presentations of distinct repeated syllables. In a first experiment, the perceptual stability of reported auditory percepts was significantly modulated by the modality presentation. In a second experiment, when presenting audiovisual stimuli consisting in a stable audio track dubbed with a video track alternating congruent and incongruent stimuli, a strict correlation between the timing of perceptual transitions and the timing of video switches was found. Finally, a third experiment showed that the vocal tract opening onset event provided by the visual input could play the role of a bootstrap mechanism in the search for transformations. Altogether, these results demonstrate the capacity of visual information to control the multistable perception of speech in its phonetic content and temporal course. The Verbal Transformation Effect thus provides a useful experimental paradigm to explore audiovisual interactions in speech perception. The ability of vision to influence verbal transformations in their temporal course raise general questions about how and when the human brain integrates input from different senses. The recent neurophysiological evidence in favour of a cortical " dorsal route " (Hickok & Poeppel, 2000, 2004) for speech perception, linking temporal, parietal and frontal regions, and involved both in audiovisual speech perception (Callan et al., 2003; Skipper et al., 2005, 2007) and in verbal transformations (Sato et al., 2004) provides a natural circuit for audiovisual verbal transformations. The fact that this circuit connects perceptual and motor regions, and that motor processes have been advocated to play a role both in the verbal transformation circuit (Sato et al., 2006) and for audiovisual integration in speech perception) suggests that such processes are indeed involved in multisensory verbal transformations.